[feature] Multi-repo workflow #13

Ranger-X · 2025-03-31T08:57:36Z

multi-repo CI workflow differs from basic CI (which in templates) in the following key aspects:

In multi-repo workflow we can push to dev and prod registries separately with their own rules (see jobs/multi-repo and/or examples/multi-repo-module.gitlab-ci.yml for example jobs).
All werf's caches and other artifacts (from build stage) are stored in Gitlab's module's registry by default. And only final images are pushed to the dev/prod registries. So, even in dev-registry there should be no "build-time garbage" and/or some "extra" images/layers for each module.

… will execute only when requested by some job Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

…w does not work when included via `remote` keyword :-( Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

examples/multi-repo-module.gitlab-ci.yml

KraMorK · 2025-03-31T12:02:53Z

jobs/multi-repo/Deploy_DEV.gitlab-ci.yml

+    MODULES_MODULE_TAG: pr${CI_MERGE_REQUEST_IID}
+  rules:
+    # do not run if some required variables is empty
+    - if: '$DEV_MODULES_REGISTRY == null || $DEV_MODULES_REGISTRY == "" || $DEV_MODULES_REGISTRY_PATH == null || $DEV_MODULES_REGISTRY_PATH == ""'


only, when and rules don't work well together. If we implement rules, then it's better to get rid of when without rules in the job and only in these jobs.

In multi-repo templates (templates/multi-repo/*.yml) I use rules everywhere. Do you mean get rid of only in simple templates (templates/*.yml)?

KraMorK · 2025-03-31T12:08:20Z

templates/multi-repo/Setup.gitlab-ci.yml

+
+.setup:
+  before_script:
+    # Setup trdl


#in gitlab-ci it is better to use docker-executor, separately prepare the image and call it. this will reduce the busy time of runners, add isolation between jobs, reduce the number of used dependencies (possible points of failure) and increase security. but perhaps they will not agree with me

I agree about a separate image for docker/kubernetes-executor.
But minor/patch versions of Werf/DMT come out quite often and maintaining a separate CI-image can be "more expensive" in terms of time than installing current versions of tools every time pipeline is launched. 😇

I can also be wrong with my conclusions. 😅

…eady merged into `main` Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

… by 10) runs before `build` stage Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

…kout submodules Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

…eploy jobs * add $DEBUG_CI_DRY_RUN variable for dry run production-related jobs Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

…istry variables (MODULES_REGISTRY_*) Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

…irepo-workflow

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* * add sign images code from `feat/signing_images` branch Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * bump WERF_SIGN_VERSION Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * try download werf from fox with CI_JOB_TOKEN Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * try download werf from fox with CI_JOB_TOKEN Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * try download werf from fox with WERF_SIGN_PACKAGE_TOKEN Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * fix werf install Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * debug Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * remove debug statements Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * disable ELF files signing Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * require WERF_VAULT_AUTH_ROLE_ID or WERF_VAULT_AUTH_JWT Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * require WERF_VAULT_AUTH_ROLE_ID or WERF_VAULT_AUTH_JWT Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * add some emojis Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> * * do not enable images signing (comment WERF_SIGN_VERSION var) by default Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com> --------- Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Ranger-X added 12 commits March 25, 2025 17:21

* default's before_script block in Setup moved to .setup, so it's…

8fb624e

… will execute only when requested by some job Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add .lint job to lint stage

d76af77

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add workflow D8.Module

40fdc7e

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* revert back to dedicated files instead of workflow, because workflo…

179c7b0

…w does not work when included via `remote` keyword :-( Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add .publish job to deploy stage

c9d8ddb

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

fix: remove dependencies: .build from .publish

8d9fccb

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add Deploy_DEV job

7a12eb9

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix .default_rules

f23c336

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix .default_rules

e888650

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add jobs/Deploy_PROD

44afc49

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

refactor: move my multi-repo version to templates/multi-repo

d40fc2e

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* multi-repo readme and example

34cad56

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Ranger-X self-assigned this Mar 31, 2025

raabdullaev requested review from KraMorK, RomanenkoDenys, Taior and nabokihms March 31, 2025 09:29

KraMorK reviewed Mar 31, 2025

View reviewed changes

* fix URL to templates in multi-repo-module.gitlab-ci.yml as it alr…

13fb130

…eady merged into `main` Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Taior approved these changes Apr 1, 2025

View reviewed changes

Ranger-X added 10 commits April 1, 2025 14:45

* add Publish default branch to DEV job to Deploy_DEV

f3eb923

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* added Cleanup job

e13130d

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add cleanup include to multi-repo example

36be3a7

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add Auto cleanup job which randomly (if current second is divided…

d7bf265

… by 10) runs before `build` stage Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix auto cleanup job rules

66f58d1

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* allow auto cleanup job to fail

ac682ef

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* refactor PROD deploy with EDITION and parallel.matrix.RELEASE_CHANNEL

f6526e5

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* do not run auto-cleanup when tag defined (release workflow)

d37be2d

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix .deploy_prod_rules

faeaae9

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add deploy to PROD EE

615652d

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Ranger-X and others added 30 commits April 25, 2025 13:39

* slightly rename deploy jobs

89dd366

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* slightly rename deploy jobs

c92fc2d

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* slightly rename deploy jobs

61fee16

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* do not download base_images.yml if BASE_IMAGES_VERSION is empty

9d381c3

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* use BASE_IMAGES_VERSION v0.5.2

1accb07

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix ROOT_VERSION for dmt's trdl repo

622482f

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix ROOT_VERSION for dmt's trdl repo

36b17a0

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add "Login to target registry"

e82b990

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add printenv for debug .publish stage

69cacca

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* remove debug

c4ef564

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Merge branch 'main' into i-makeev/feature-multirepo-workflow

6070ca6

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

WIP: add Svace setup/init to templates/multi-repo/Setup.gitlab-ci.yml

dd36df0

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* remove werf managed-images ls from Scheduled cleanup task

55e33e7

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* apply private git repo patch before werf cleanup to properly chec…

a668412

…kout submodules Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* increase Scheduled cleanup timeout to 3 hour

e15f278

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* add "Logging in to registry ..." for easy debug login issues

8b30a26

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* separate MODULES_REGISTRY and MODULES_TARGET_REGISTRY for publish/d…

e62bd77

…eploy jobs * add $DEBUG_CI_DRY_RUN variable for dry run production-related jobs Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* set default for MODULES_TARGET_REGISTRY_* variables from source reg…

df08fdf

…istry variables (MODULES_REGISTRY_*) Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* set default for MODULES_TARGET_REGISTRY_* variables from source reg…

8e962ff

…istry variables (MODULES_REGISTRY_*) Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* get templates/CVE_Scan from main branch

ec2c5e9

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

[CVE_Scan] feat: generate docker config without docker login

053c4cc

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* deploy tags also to dev-registry when tag is specified

5c9c3c9

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix DEV | Publish tags also to dev-registry job

282554e

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Merge remote-tracking branch 'origin/main' into i-makeev/feature-mult…

6b81bb4

…irepo-workflow

* add Svace integration from main branch

d56add5

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* fix DECKHOUSE_LIB_HELM_VERSION block indent in multi-repo/Setup

4f63159

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* get CVE_Scan from v8.0 branch and patch to auth without docker

30c8df3

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Merge branch 'main' into i-makeev/feature-multirepo-workflow

ef6b324

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

* show WERF_REPO on cleanup

a21918e

Signed-off-by: Ivan.Makeev <ivan.makeev@flant.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[feature] Multi-repo workflow #13

[feature] Multi-repo workflow #13

Uh oh!

Ranger-X commented Mar 31, 2025

Uh oh!

Uh oh!

KraMorK Mar 31, 2025

Uh oh!

Ranger-X Apr 1, 2025

Uh oh!

KraMorK Mar 31, 2025

Uh oh!

Ranger-X Apr 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[feature] Multi-repo workflow #13

Are you sure you want to change the base?

[feature] Multi-repo workflow #13

Uh oh!

Conversation

Ranger-X commented Mar 31, 2025

Uh oh!

Uh oh!

KraMorK Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

Ranger-X Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

KraMorK Mar 31, 2025

Choose a reason for hiding this comment

Uh oh!

Ranger-X Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants